Towards Better Quality Specification Miners

Softwares are often built without specification. Tools to automatically extract specification from software are needed and many techniques have been proposed. One type of these specifications – temporal API specification – is often specified in the form of automaton (i.e., FSA/PFSA). There have been many work on mining software temporal specification using dynamic analysis techniques; i.e., analysis of software program traces. Unfortunately, the issues of scalability, robustness and accuracy of these techniques have not been comprehensively addressed. In this paper, we describe a framework that enables assessments of the performance of a specification miner in generating temporal specification of software through traces recorded from its API interaction. Our framework requires the temporal specification produced by the miner to be expressed as probabilistic finite state automaton (PFSA). The framework accepts a user-defined simulator PFSA and a specification miner. It produces quality assurance measures on the specification generated by the miner. We investigate metrics used in these measures by adapting techniques found in artificial intelligence, program analysis, bioinformatics and data mining to the software specification domain. Extensive experiments on two specification miners have been performed to evaluate the effectiveness of the proposed quality assurance measures.